1. Field of the Invention
The present invention relates to an image pick-up device such as electronic still camera and digital camera including a solid state image sensing element, and more particularly to an image pick-up device which can produce an image signal having a wider dynamic range than that of a solid state image sensing element. The present invention also relates to a record medium having recorded thereon a computer readable program for controlling an imaging operation of the above mentioned image pick-up device.
2. Description of the Related Art
In the image pick-up device of the kind mentioned above, since the solid state image sensing element has a relatively narrow dynamic range compared with a conventional photographic film, when a subject situating on a relatively bright background is to be picked-up under such an exposure time that the subject is picked-up with a proper exposure amount, the background might be picked-up with an overexposure amount and an image signal corresponding to the background might be saturated. On the other hand, when the subject is picked-up under such an exposure time that the bright background is picked-up with a proper exposure amount, the subject might be picked-up with an underexposure amount and might become too dark. In order to increase a dynamic range of the solid state image sensing element, it has been proposed a known image pick-up device, in which an object is picked-up twice with different exposure amounts, and the thus obtained two image signals are synthesized by simply adding these image signals each other or by partially exchanging or combining these image signals in a mosaic fashion.
In the former solution in which the two image signals obtained with different exposure amounts are synthesized by simply adding them, when a subject moves or a camera moves to produce a relative positional shift between successively picked-up two image signals, S/N is decreased at a portion which is subjected to the movement and pseudo-color or pseudo-edge might be produced in a synthesized image signal.
In order to remove such a problem, in Japanese Patent Application Laid-open Publication Kokai Hei 2-280585, it has been proposed to derive a synthesized image signal in such a manner that a difference between the two image signals is derived and an absolute value of said difference is subtracted from the synthesized image signal obtained by merely adding the two image signals. However, in this case, since the image shift portion in the synthesized image signal is removed by the subtractive correction, when a subject moves, a configuration of the subject within a synthesized image might be deformed, and therefore the synthesized image has a feeling of strangeness as compared with an image formed on a conventional photographic film.
The above problem also occurs in the later solution, in which the two image signals are synthesized by combining or exchanging them in a mosaic fashion. That is to say, as shown in FIG. 13, when a too bright portion B (e.g. background) in a first image A picked-up with a large exposure amount is replaced by a corresponding portion D in a second image C picked-up with a small exposure amount after adjusting a gain of said portion D to derive a synthesized image E, if a subject F moves from right to left in FIG. 13 during a time interval between successive image picking-up operations, a part of the subject F in the second image C picked-up with small exposure amount corresponding to the movement is also synthesized as a part G in the synthesized image E. Then, the subject is partially seen double, and therefore a feeling of strangeness remains.
A first object of the present invention is to provide a novel and useful image pick-up device, in which a synthesized image having a wider dynamic range than that of a solid state image sensing element can be obtained by partially combining or exchanging a plurality of images picked-up with different exposure amounts in a mosaic fashion, while said synthesized image be seen without causing a feeling of strangeness.
A second object of the invention is to provide a record medium having recorded thereon a computer readable program which is to make a computer to execute procedures for controlling the image pick-up device in such a manner that a synthesized image with a wider dynamic range than that of a solid state image sensing element can be seen without causing a feeling of strangeness.
According to the invention for attaining the above mentioned first object, an image pick-up device for producing a synthesized image signal having a wide dynamic range by partially combining a plurality of images picked-up with different exposure amounts comprises:
an image picking-up means including a solid state image sensing element and picking-up an object by a plurality of times with different exposure amounts to derive a plurality of image signals;
a memory means for storing at least a part of said plurality of image signals derived successively from said image picking-up means;
an image synthesizing means for synthesizing said plurality of image signals a part of which is read out of said memory means to derive a synthesized image signal;
a detecting means for detecting a synthetic unsuitable portion in said plurality of image signals which is not suitable for synthesizing in accordance with a comparison of the plurality of image signals;
a correcting means for correcting pixel signals of said synthetic unsuitable portion detected by said detecting means to derive a corrected pixel signal; and
a combining means for combining said corrected pixel signal from said correcting means with said synthesized image signal from said synthesizing means to derive a corrected synthesized image signal.
In the image pick-up device according to the invention, a synthetic unsuitable portion is detected on the basis of a comparison of a plurality of image signals picked-up with different exposure amounts, and the pixel signal of the thus detected synthetic unsuitable portion is corrected. Therefore, even if a subject moves on a bright background during successive image picking-up operations, it is possible to minimize any undesired synthesis due to the movement of the subject, and thus a corrected synthesized image having a wide dynamic range can be obtained without causing a feeling of strangeness upon comparing with the conventional photographic film.
In an embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed such that a ratio in level between a plurality of image signals picked-up with different exposure amounts is calculated, and a synthetic unsuitable portion is detected on the basis of a comparison of said ratio with a ratio in exposure amounts with which said plurality of image signals are picked-up. In this case, a synthetic unsuitable portion can be detected by a simple comparing operation using the plurality of image signals picked-up with different exposure amounts and data from a hardware controlling an exposing operation. Therefore, a cost can be reduced and an operation time can be shortened.
In another embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed such that a difference between a plurality of image signals picked-up with different exposure amounts is calculated, and a synthetic unsuitable portion is detected on the basis of said difference. Also in this case, a synthetic unsuitable portion can be detected by a simple comparing operation using the plurality of image signals taken with different exposure amounts and data from a hardware controlling an exposing operation, and therefore a cost can be reduced and an operation time can be shortened.
In an embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed such that a motional vector is calculated on the basis of a comparison of the plurality of image signals picked-up with different exposure amounts, and a synthetic unsuitable portion is detected on the basis of a magnitude of said motional vector. In this embodiment, since a moving amount and a moving direction of a subject can be derived, it is possible to detect the synthetic unsuitable portion much more precisely.
In an embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed such that a difference between said plurality of image signals picked-up with different exposure amounts is calculated, a motional vector is derived on the basis of a comparison between said plurality of image signals only for a pixel region in which an absolute value of said difference is not less than a predetermined value, and a synthetic unsuitable portion is detected on the basis of a magnitude of said motional vector. When a synthetic unsuitable portion is detected by calculating a motional vector only for a pixel region in which an absolute value of a difference between a plurality of image signals picked-up with different exposure amounts in the manner mentioned above, the operation time of detecting a synthetic unsuitable portion can be materially shortened.
In an embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion correcting means is constructed such that a pixel signal within a synthetic unsuitable portion is corrected in accordance with a synthesized pixel signal of a portion which situates in a portion which is not detected as the synthetic unsuitable portion, but is in a vicinity of the synthetic unsuitable portion. When the pixel signal within the synthetic unsuitable portion is corrected in accordance with the synthesized pixel signal of a portion which situates near the synthetic unsuitable portion, it is possible to obtain a corrected synthesized image signal which does not produce a feeling of strangeness with respect to surroundings of the subject as well as a whole synthesized image signal.
In another embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed to detect a synthetic unsuitable portion on the basis of the motional vector, and said synthetic unsuitable portion correcting means is constructed such that a pixel signal in the synthetic unsuitable portion including a start point of the motional vector is corrected by a synthesized pixel signal in a portion including an end point of said motional vector and being excluded from said synthetic unsuitable portion, and a pixel signal of the synthetic unsuitable portion in a remaining region within the pixel block including the end point of the motional vector is corrected in accordance with an image signal of a corresponding region of one of said plurality of image signals. In this manner, when the pixel signal in the synthetic unsuitable portion including the start point of the motional vector is corrected by the synthesized pixel signal in the pixel block including the end point of said motional vector and being excluded from said synthetic unsuitable portion, the pixel signal in the synthetic unsuitable portion including the start point of the motional vector can be corrected using a lot of pixels without the synthetic suitable portion. Therefore, the precision of the correction can be improved, and it is possible to attain a corrected synthesized image which does not cause a feeling of strangeness with respect to a whole image.
In an embodiment of the image pick-up device according to the invention, said synthetic unsuitable portion detecting means is constructed such that a synthetic unsuitable portion is detected by calculating a motional vector, and said synthetic unsuitable portion correcting means is constructed such that a pixel signal of the synthetic unsuitable portion within a pixel block which has a center at a start point of said motional vector and contains an end point of said motional vector is corrected in accordance with a synthesized image signal of a synthetic suitable portion within a pixel block which has a center at the end point of said motional vector and has a same size as said pixel block having a center at the start point of the motional vector. Then, it is no more necessary for correcting the pixel signal of the synthetic unsuitable portion to perform a calculation at a pixel unit, and thus the structure can be simplified and the operating time can be shortened.
In order to achieve the second object of the invention, a computer readable record medium, having a program recorded thereon, wherein said program is to make a computer execute the following procedures for controlling an operation of an image pick-up device having a solid state image sensing element:
to derive a plurality of image signals with different exposure amounts from said solid state image sensing element;
to detect a synthetic unsuitable portion on the basis of a comparison of said plurality of image signals;
to correct a pixel signal of said detected synthetic unsuitable portion to produce a corrected pixel signal;
to synthesize said plurality of image signals with each other except for said synthetic unsuitable portion to derive a synthesized image signal; and
to combine said synthesized image signal with said corrected pixel signal of the synthetic unsuitable portion to derive a corrected synthesized image signal having a dynamic range which is wider than that of said solid state image sensing element.